In accordance with the development of computer graphics and the like, there has been intensive efforts to provide a system for practical usage in three-dimensional graphics. However, one appreciable problem in accordance with the spread of such a system of practical usage is the method of obtaining shape data. More specifically, the task of entering the complicated three-dimensional shape of an object having a free-form surface or that resides in the natural world into a computer is extremely tedious and difficult.
Furthermore, in reconstructing an object with a computer and the like, it is difficult to express the texture of the surface of the object in a more realistic manner by just simply reconstructing the shape of the object.
Three-dimensional image information can be handled more easily if the shape information and color/texture information can be reconstructed within the computer based on image information that is obtained by shooting an actual object.
In three-dimensional image communication such as by, for example, the Internet, the opportunity of a general user to create a three-dimensional image who is the transmitter of information will increase. Therefore, the need arises for a simple and compact apparatus that produces a three-dimensional image.
(1) Japanese Patent Laying-Open No. 5-135155 discloses a three-dimensional model generation apparatus that can construct a three-dimensional model from a series of silhouette images of an object of interest placed on a turntable under the condition of normal illumination.
According to this three-dimensional model construction apparatus, an object of interest that is rotated on a turntable is continuously shot by a camera. The silhouette image of the object of interest is extracted from the obtained image by an image processing computer. By measuring the horizontal distance from the contour of the silhouette image to the vertical axis of rotation for the silhouette image, a three-dimensional model is generated according to this horizontal distance and the angle of rotation. More specifically, the contour of the object of interest is extracted from the continuously shot silhouette images to be displayed as a three-dimensional model.
FIG. 1 is a diagram representing the concept of assigning texture information to the three-dimensional model generated as described above according to the image information continuously picked up by a camera.
Japanese Patent Laying-Open No. 5-135155 discloses the case of obtaining image information by continuously rotating an object of interest and shooting the same, i.e., obtaining image information in the resolution level of shape recognition with respect to a three-dimensional model of a human figure. More specifically, an image is picked up for every 1° of rotation to obtain 360 images with respect to the object of interest.
For the sake of simplifying the description, the case of shooting an image for every larger stepped angle will be described hereinafter. However, the essence is identical.
Consider the case of picking up a total of n images by rotating an object of interest for every predetermined angle of rotation, as shown in FIG. 1. In this case, each image information corresponds to the label number of 1, 2, 3 . . . , n.
The object of interest is represented as a shape model (wire frame model) 300 using a polygon (triangular patch). When texture information is to be assigned to shape model 300, color information (texture information) of the image information of a corresponding label number is assigned for each triangular patch according to the direction of the camera shooting the object of interest.
More specifically, based upon the vector towards the target triangular patch from the axis of rotation of shape model 300, the texture information with respect to the triangular patch is captured from the image that has the direction of the shooting direction vector and this vector most closely matched. Alternatively, from the standpoint of intuition, a plurality of lines such as the circles of longitude of a terrestrial globe can be assumed with respect to the surface of the model. Texture information can be captured from the first image information for the triangular patch in the range of 0° to 1×360/n°, from the second image information for the triangular patch in the range of 1×360/n° to 2×360/n°, and so on. This method of capturing texture information will be referred to as the central projection system hereinafter.
The central projection system is advantageous in that image information can be provided in a one-to-one correspondence with respect to each triangular patch or the constituent element forming the shape model (referred to as “three-dimensional shape constituent element” hereinafter), and that this correspondence can be determined easily.
However, the central projection system is disadvantageous in that the joint of the texture is noticeable when the gloss or the texture of the color information is slightly different due to the illumination and the like since the texture information is assigned from different image information (image information of a different label number) to a three-dimensional shape constituent element that is not present within the same range of rotation angle when viewed from the axis of rotation.
Furthermore, a corresponding three-dimensional shape constituent element may be occluded in the image information obtained from a certain direction of pickup depending upon the shape of the object of interest. There is a case where no texture information corresponding to a certain three-dimensional shape constituent element is included in the corresponding image information.
FIG. 2 is a diagram for describing such a situation. In FIG. 2, the relationship is shown of the axis of rotation, the cross section of the object of interest and the object image projected in the camera at a vertical plane including the axis of rotation of the object of interest. When the object of interest takes a shape that has an occluded region that cannot be viewed from the camera as shown in FIG. 2, the image information picked up from this angle direction is absent of the texture information corresponding to this occluded region. However, texture information of this occluded region can be captured from another pickup direction that has a certain angle with respect to the previous direction of pickup.
(2) As a conventional method, extraction of an object portion from an image of an object can be effected manually using an auxiliary tool. More specifically, the image of an object obtained by shooting the target object together with the background is divided into a plurality of regions. The operator selects the background area in the image of the object to erase the background area using a mouse or the like. However, this method is disadvantageous in that the burden on the operator for the manual task is too heavy.
Another conventional method of object extraction employs the chroma-key technique. More specifically, the portion of the object is extracted from the image of the object using a backboard of the same color. However, this method is disadvantageous in that a special environment of a backboard of the same color has to be prepared.
A further conventional method of object extraction employs the simple difference method. More specifically, difference processing is effected between an object image and a background image in which only the background of the object of interest is shot to obtain the difference. The area having an absolute value of the difference greater than the threshold value is extracted as the portion of the object. However, there is a problem that, when the object of interest includes an area of a color identical to the color of the background, that portion cannot be extracted as a portion of the object. In other words, this method is advantageous in that the extraction accuracy of the object portion is poor.
Another conventional method of object extraction takes advantage of the depth information by the stereo method. More specifically, the area with the depth information that is smaller than a threshold value is extracted as the portion of an object of interest from an image of the object obtained by shooting the object together with the background. However, the difference in depth is so great in the proximity of the boundary between the object of interest and the background that proper depth information cannot be obtained reliably. There is a problem that a portion of the background is erroneously extracted as a portion of the object.
All of the above-described conventional methods require the determination of a threshold value in advance. It is extremely difficult to determine an appropriate threshold value on account of the conversion. property of the A/D converter for converting the image and the property of the illumination. There is also the problem that the threshold value must be reselected when the conversion characteristic of the A/D converter or the property of the illumination is changed.
(3) A three-dimensional digitizer is known as a conventional apparatus of reading out the shape of an object of interest. The three-dimensional digitizer includes an arm with a plurality of articulations and a pen. The operator provides control so as to bring the pen in contact with the object of interest. The pen is moved along on the object of interest. The angle of the articulation of the arm varies as the pen is moved. A three-dimensional shape of the object of interest is obtained according to the angle information of the articulation of the arm. However, such a digitizer is disadvantageous in that the time and the labor of the task of measurement by manual means are too great and heavy.
The laser scanner is known as another conventional apparatus. The laser scanner directs a laser beam on an object of interest to scan the object. As a result, a three-dimensional shape of the object of interest is obtained. There is a problem that a three-dimensional model of an object of interest formed of a substance that absorbs light cannot be obtained with such a laser scanner. There is also the problem that the apparatus is extremely complex and costly. Furthermore, there is a problem that the environment for pickup is limited since measurement of the object of interest must be carried out in a dark room. There is also the problem that color information cannot be easily input.
U.S. Pat. No. 4,982,438 discloses a three-dimensional model generation apparatus. This apparatus computes a hypothetical existing region using the silhouette image of an object of interest. This hypothetical existing region is a conical region with the projection center of the camera as the vertex and the silhouette of an object of interest as the cross section. This conical region (hypothetical existing region) is described with a voxel model. This process is carried out for a plurality of silhouette images. Then, a common hypothetical existing region is obtained to generate a three-dimensional model of the object of interest. Here, the common hypothetical existing region is the ANDed area of a plurality of hypothetical existing regions with respect to the plurality of silhouette images. However, there is a problem that a three-dimensional model of high accuracy cannot be generated when there is one inaccurate silhouette image since the three-dimensional shape is obtained by the AND operation. There is also a problem that color information is insufficient or a local concave area cannot be recognized since the object of interest is shot only from a horizontal direction (direction perpendicular to the axis of rotation).
In the above three-dimensional model generation apparatus of Japanese Patent Laying-Open No. 5-135155, an object of interest that is rotating on a turntable is shot by a camera to obtain a plurality of silhouette images. A plurality of shapes of the object of interest at a plurality of horizontal planes (a plane perpendicular to the axis of rotation) are obtained on the basis of these plurality of silhouette images. The points on the contour line of the shape of the object of interest in adjacent horizontal planes are connected as a triangular patch. The point on the contour line of the shape of the object of interest in one horizontal plane is determined for every predetermined angle. A three-dimensional model of an object of interest is generated in this way. However, there is a problem in this apparatus that a special environment for shooting is required since a backboard to generate a silhouette image is used. Furthermore, the amount of data is great since the three-dimensional model is generated using the shape of the object of interest in a plurality of horizontal planes. There was a problem that the process is time consuming.
In view of the foregoing, an object of the present invention is to provide a method and apparatus of texture information assignment that allows assignment of texture information to each three-dimensional shape constituent element forming a shape model regardless of the shape of the object of interest in the event of reconstructing a three-dimensional model within a computer and the like according to image information obtained by shooting a real object.
Another object of the present invention is to provide a method and apparatus of texture information assignment that allows assignment of texture information approximating the texture of a real object from image information obtained by shooting a real object in the assignment of texture information to a shape model according to picked up image information.
A further object of the present invention is to provide a method and apparatus of texture information assignment with less noticeable discontinuity (seam) in texture assigned to each three-dimensional shape constituent element constructing a shape model in assigning texture information to the shape model according to image information obtained by shooting a real object.
Still another object of the present invention is to provide a method and apparatus of object extraction that allows a portion, if present, of an object of image having a color identical to that of the background extracted.
A still further object of the present invention is to provide a method and apparatus of object extraction that can extract always stably and properly a portion of an object even when various characteristics change.
Yet a further object of the present invention is to provide a method and apparatus of object extraction that can have manual task reduced, and dispensable of a special shooting environment.
Yet another object of the present invention is to provide a method and apparatus of three-dimensional model generation that can have manual task reduced.
Yet a still further object of the present invention is to provide a method and apparatus of three-dimensional model generation of a simple structure with few limitation in the shooting environment and substance of the object of interest.
An additional object of the present invention is to provide a method and apparatus of three-dimensional model generation that can generate a three-dimensional model in high accuracy even if there are several inaccurate ones in a plurality of silhouette images.
Still a further object of the present invention is to provide a method and apparatus of three-dimensional model generation in which sufficient color information can be obtained and that allows recognition of a local concave portion in an object of interest.
Yet a still further object of the present invention is to provide a method and apparatus of three-dimensional model generation that can generate a three-dimensional model at high speed with fewer data to be processed, dispensable of a special shooting environment.